Goto

Collaborating Authors

 metadata file


Responsible LLM Deployment for High-Stake Decisions by Decentralized Technologies and Human-AI Interactions

arXiv.org Artificial Intelligence

High-stakes decision domains are increasingly exploring the potential of Large Language Models (LLMs) for complex decision-making tasks. However, LLM deployment in real-world settings presents challenges in data security, evaluation of its capabilities outside controlled environments, and accountability attribution in the event of adversarial decisions. This paper proposes a framework for responsible deployment of LLM-based decision-support systems through active human involvement. It integrates interactive collaboration between human experts and developers through multiple iterations at the pre-deployment stage to assess the uncertain samples and judge the stability of the explanation provided by post-hoc XAI techniques. Local LLM deployment within organizations and decentralized technologies, such as Blockchain and IPFS, are proposed to create immutable records of LLM activities for automated auditing to enhance security and trace back accountability. It was tested on Bert-large-uncased, Mistral, and LLaMA 2 and 3 models to assess the capability to support responsible financial decisions on business lending.


Column Vocabulary Association (CVA): semantic interpretation of dataless tables

arXiv.org Artificial Intelligence

Traditional Semantic Table Interpretation (STI) methods rely primarily on the underlying table data to create semantic annotations. This year's SemTab challenge introduced the ``Metadata to KG'' track, which focuses on performing STI by using only metadata information, without access to the underlying data. In response to this new challenge, we introduce a new term: Column Vocabulary Association (CVA). This term refers to the task of semantic annotation of column headers solely based on metadata information. In this study, we evaluate the performance of various methods in executing the CVA task, including a Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) approach, as well as a more traditional similarity approach with SemanticBERT. Our methodology uses a zero-shot setting, with no pretraining or examples passed to the Large Language Models (LLMs), as we aim to avoid a domain-specific setting. We investigate a total of 7 different LLMs, of which three commercial GPT models (i.e. gpt-3.5-turbo-0.125, gpt-4o and gpt-4-turbo) and four open source models (i.e. llama3-80b, llama3-7b, gemma-7b and mixtral-8x7b). We integrate this models with RAG systems, and we explore how variations in temperature settings affect performances. Moreover, we continue our investigation by performing the CVA task utilizing SemanticBERT, analyzing how various metadata information influence its performance. Initial findings indicate that LLMs generally perform well at temperatures below 1.0, achieving an accuracy of 100\% in certain cases. Nevertheless, our investigation also reveal that the nature of the data significantly influences CVA task outcomes. In fact, in cases where the input data and glossary are related (for example by being created by the same organizations) traditional methods appear to surpass the performance of LLMs.


PyTorch Multi-Weight Support API Makes Transfer Learning Trivial Again

#artificialintelligence

To have a solid understanding of what is happening, we will examine the old ways first. We won't train a model, but we'll do almost everything else: You set the pretrained flag to True to tell PyTorch that you do not want it to initialize the model's weights randomly. Instead, it should use the weights obtained by training the model on the ImageNet dataset. Then, you define and initialize a composition of data transformations. Next, you are ready to process the image and pass it through the neural network layers to get your output.


Fundamentals of Data Versioning You Must Know

#artificialintelligence

Fundamentally, every possible way of changing datasets and the way we process our datasets (which naturally involves changing our code) represents an "experiment", and we want to keep track of every "experiment" we do. We need to manage versions of the data that were used to TRAIN, VALIDATE, and TEST ML models along with the ML models themselves. Data versioning means recording a specific moment across the evolution of data through a specific version number. This process in machine learning is valuable because the necessity of rolling back to a specific situation that brought us to the creation of a specific model cannot be overstated. Briefly and practically, to be able to reconstruct our project to a specific point in time, we must maintain the record of three objects: The code, the data, and the model.


Using machine learning services

#artificialintelligence

This demo will show you how to leverage a machine learning service by running a churn model created externally and using its outputs in Pega Prediction Studio. The steps are similar to using other machine learning services such as Google AI Platform. Using a machine learning service instead of a model that runs locally may involve costs and possible down time of the service. However, for certain use cases such as churn or credit risk models, machine learning services can be the optimal choice. To showcase how to use a churn model created in Amazon SageMaker, let's first consider the high-level steps involved in creating a machine learning model.


How AI and metadata are taking the hard work out of content discovery

#artificialintelligence

This is a particularly tough time for broadcasters and service providers. There's more competition than ever before thanks to instant streaming and on-demand viewing, and each company is in a battle for the best content. Pay TV operators have built enormous on-demand catalogues, and broadcasters are expanding their online services with more library content and short-form video. The ultimate goal for each company is to draw in as many viewers as possible, and ultimately to keep them engaged for as long as possible, too. But, with an ever-expanding sea of content in front of them, it's getting more difficult for viewers to choose what they want to watch.